Comments for MEDB 5501, Week 8

Review multiple comparisons issue

  • Type I error: rejecting the null hypothesis when the null hypothesis is true.
    • Multiple simultaneous hypotheses increase the Type I error rate.

Bonferroni inequality for two simultaneous hypotheses

Define

  • \(E_1\) = Type I error for Hypothesis 1
  • \(E_2\) = Type I error for Hypothesis 2
    • \(P[E_1\ \cup\ E_2]=P[E_1]+P[E_2]-P[E_1\ \cap\ E_2]\)
    • \(P[E_1\ \cup\ E_2]\ \le\ P[E_1]+P[E_2]\)
    • \(P[E_1\ \cup\ E_2]\ \le\ P[E_1]+P[E_2]\)
    • \(P[E_1\ \cup\ E_2]\ \le\ 2 \alpha\)

Bonferroni adjustment

  • For m hypotheses
    • \(P[E_1\ \cup\ ...\ \cup E_m]\ \le\ m \alpha\)
  • Test each hypothesis at \(\alpha/m\)
    • Preserves overall Type I error rate
  • Example, 3 simultaneous hypotheses
    • Reject H0 if p-value < 0.0133

Controversies over Bonferroni adjustment

  • Increases Type II errors
  • Impractical for large values of m
  • Works poorly for highly correlated tests
  • Ambiguity in definition of “simultaneous” hypotheses

Alternatives to Bonferroni adjustment

  • False discovery rate
  • Designation of primary and secondary outcomes
  • Subjective assessment of simultaneous hypotheses

Review two-sample t-test

  • \(H_0:\ \mu_1=\mu_2\)
  • \(H_1:\ \mu_1 \ne \mu_2\)
    • Or a one tailed alternative
  • \(T=\frac{\bar X_1-\bar X_2}{SE(\bar X_1-\bar X_2)}\)
    • Accept H0 if T is close to zero.

What to do with three or more groups?

  • \(H_0:\ \mu_1=\mu_2=...=\mu_k\)
  • \(H_1:\ \mu_i \ne \mu_j\) for some i, j
    • Note: one-tailed test is tricky.
  • Accept H0 if the F ratio (defined below) is close to 1.

Important assumptions

  • Same as independent-samples t-test
    • Normality
    • Equal variances
    • Independence

How to check assumptions

  • Boxplots
  • Analysis of residuals, \(e_{ij}\)
    • \(e_{ij}\)= Observed - Predicted
    • \(e_{ij}= Y_{ij}-\bar{Y}_i\)

Tukey post hoc tests

  • If you reject H0, which values are unequal
    • With k groups, there are k(k-1)/2 comparisons
  • Studentized range (Tukey test)
    • Requires equal sample sizes per group
    • Uses harmonic mean approximation for unequal sample sizes.
      • Do not use harmonic means if seriously different sample sizes.

Alternatives to Tukey post hoc tests

  • Bonferroni adjustment
    • Works for unequal sample sizes per group
    • Works for unequal variances
  • Dunnett’s test
    • Treatment versus multiple controls
  • Scheffe’s test
    • Works for complex comparison
      • Example \(\mu_1\ vs.\ \frac{\mu_2+\mu_3+\mu_4}{3}\)

Artificial data

   g  y
1  1 23
2  1 30
3  1 25
4  2 33
5  2 36
6  2 41
7  2 37
8  2 43
9  3 24
10 3 29
11 3 31

Scatterplot

Total SS = 452

Within SS = 116

Between SS = 336

Degrees of freedom

  • For Total SS, df = 10
  • For Within SS, df = 8
  • For Between SS, df = 2
  • In general,
    • N = number of observations
    • k = number of groups
      • Total df = N-1
      • Within df = N-k
      • Between df = k-1

ANOVA table, 1 of 4

ANOVA table, 2 of 4

ANOVA table, 3 of 4

ANOVA table, 4 of 4

ANOVA table from the general linear model

Regression approach to ANOVA, 1 of 4

Regression approach to ANOVA, 2 of 4

Regression approach to ANOVA, 3 of 4

Regression approach to ANOVA, 4 of 4

Predicted values and residuals, 1 of 2

Predicted values and residuals, 2 of 2

What belongs in an interpretation

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

  • Lang and Secic, How to Report Statistics in Medicine, p110.

Identify the dependent variable

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

  • Lang and Secic, How to Report Statistics in Medicine, p110.

Summarize each group

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

  • Lang and Secic, How to Report Statistics in Medicine, p110.

Explain how subjects were assigned

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

  • Lang and Secic, How to Report Statistics in Medicine, p110.

Describe the statistical test

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

  • Lang and Secic, How to Report Statistics in Medicine, p110.

Specify the alpha level

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.

  • Lang and Secic, How to Report Statistics in Medicine, p110.

Specify the F statistic with degrees of freedom

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

  • Lang and Secic, How to Report Statistics in Medicine, p110.

State the p-value

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

  • Lang and Secic, How to Report Statistics in Medicine, p110.

Describe any post hoc analyses

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

  • Lang and Secic, How to Report Statistics in Medicine, p110.

Yeast activation experiment

“To shorten the time it takes him tomake his favorite pizza, a student designed an experiment to test the effect of sugar and milk on the activation times for baking yeast. Specifically, he tested four different recipes and measured how many seconds it took for the same amount of dough to rise to the top of a bowl. He randomized the order of the recipes and replicated each treatment 4 times.”

  • DASL, Activating baking yeast

Yeast analysis, descriptive statistics

Yeast analysis, ANOVA table

Yeast analysis, parameter estimates

Yeast analysis, Tukey post hoc, 1 of 2

Yeast analysis, Tukey post hoc, 2 of 2

Yeast analysis, interpretation, 1 of 3

  • The average activation times were compared using oneway ANOVA with a Tukey post hoc follow-up test.
  • All tests used a two-sided alpha level of 0.05.

Yeast analysis, interpretation, 2 of 3

  • There were statistically significant differences (p < 0.001) in the activation times of the four recipes. Recipes D and B had similar average activation times which were significantly faster than Recipe A, which itself was significantly faster than recipe C.

Yeast analysis, interpretation, 3 of 3

  • Recipes D and B were similar and had the fastest activation times. These differences were more than four minutes faster than the next best recipe which represents a practical as well as statistically significant result.

Recode into different variables dialog box

Old and new Values dialog box

General Linear Model | Univariate dialog box

Descriptives output

Boxplot

Analysis of variance table

Analysis of variance table with irrelevant rows removed

Parameter estimates

Tukey post hoc test, 1 of 7

Tukey post hoc test, 2 of 7

Tukey post hoc test, 3 of 7

Tukey post hoc test, 4 of 7

Tukey post hoc test, 5 of 7

Tukey post hoc test, 6 of 7

Tukey post hoc test, 7 of 7

Checking assumptions, boxplots

Checking assumptions, descriptive statistics

Residual analysis, 1 of 2

Residual analysis, 2 of 2

Analysis of school year, 1 of 4

Analysis of school year, 2 of 4

Analysis of school year, 3 of 4

Analysis of school year, 4 of 4